Automated camera mode selection

ABSTRACT

A camera mode to use for capturing an image or video is selected by estimating high dynamic range (HDR), motion, and light intensity with respect to a scene of the image or video to capture. An image capture device includes a HDR estimation unit to detect whether HDR is present in a scene of an image to capture, a motion estimation unit to determine whether motion is detected within the scene, and a light intensity estimation unit to determine whether a scene luminance for the scene meets a threshold. A mode selection unit selects a camera mode to use for capturing the image based on output of the HDR estimation unit, the motion estimation unit, and the light intensity estimation unit. An image sensor captures the image according to the selected camera mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/277,430, filed on Mar. 18, 2021, which is a 371 of InternationalApplication No. PCT/US2019/035686, filed on Jun. 6, 2019, which claimspriority to U.S. Provisional Application No. 62/733,308, filed on Sep.19, 2018, the entire contents of which are hereby incorporated byreference.

TECHNICAL FIELD

This disclosure relates to automated camera mode selection.

BACKGROUND

Image capture devices, such as cameras, may capture content as images orvideo. Light may be received and focused via a lens and may be convertedto an electronic image signal by an image sensor. The image signal maybe processed by an image signal processor to form an image, which may bestored and/or encoded. The configurations used by the image sensor tocapture the images or video may in some cases have an effect on theoverall quality of the images or video. For example, differentconfigurations may be used based on a particular camera mode selectedfor capturing the images or video.

SUMMARY

This disclosure describes, inter alia, systems and techniques forautomated camera mode selection.

A first aspect of this disclosure is an image capture device forautomated camera mode selection. The image capture device includes ahigh dynamic range (HDR) estimation unit configured to detect whetherHDR is present in a scene of an image to capture based on one or moredynamic range inputs, a motion estimation unit configured to determinewhether motion is detected within the scene based on one or more motioninputs, a light intensity estimation unit configured to determinewhether a scene luminance for the scene meets a thresholds based on oneor more light intensity inputs, a mode selection unit configured toselect a camera mode to use for capturing the image based on output ofthe HDR estimation unit, the motion estimation unit, and the lightintensity estimation unit, and an image sensor configured to capture theimage according to the selected camera mode. In an implementation, theoperations performed by the HDR estimation unit, the motion estimationunit, and the light intensity estimation unit are continuously performeduntil user input indicating to capture the image is received. In animplementation, the image capture device further includes a temporalsmoothing unit configured to perform temporal smoothing filteringagainst the outputs from the HDR estimation unit, from the motionestimation unit, and from the light intensity estimation unit, where themode selection unit is configured to select the camera mode based on anoutput of the temporal smoothing filtering. In an implementation, theHDR estimation unit is configured to detect whether HDR is present inthe scene of the image based on spatial information. In animplementation, the outputs of the HDR estimation unit, the motionestimation unit, and the light intensity estimation unit comprise fuzzyvalues, where the mode selection unit is configured to select the cameramode by defuzzifying the fuzzy values. In an implementation, theselected camera mode is used for capturing a second image responsive toa determination that a scene of the image is similar to a scene of thesecond image. In an implementation, the selected camera mode is used forcapturing a second image responsive to a determination that a differencebetween a first time at which the image is captured and a second time atwhich user input indicating to capture the second image is receivedmeets a threshold. In an implementation, the HDR estimation unit isconfigured to detect whether HDR is present in the scene of the imagebased on a number of dark pixels and a number of bright pixels. In animplementation, the motion estimation unit is configured to determinewhether motion is detected based on an angular speed. In animplementation, the motion estimation unit is configured to determinewhether motion is detected based on a Sum of Absolute Differences (SAD)between current and previous thumbnails when the angular speed is belowa first threshold. The first aspect may include any combination of thefeatures described in this paragraph and in the paragraphs of the secondaspect and the third aspect.

A second aspect of this disclosure is an imaging system. The imagingsystem includes a processor and an image sensor connected to theprocessor. The processor is configured to determine a high dynamic range(HDR) level in a scene of a to be captured image based on a number ofdark pixels and a number of bright pixels, determine a motion levelwithin the scene based on motion inputs, determine a scene luminancelevel based on light intensity inputs, and automatically select a cameramode based on a combination of the HDR level, the motion level, and thescene luminance level. The image sensor is configured to capture theimage according to the selected camera mode. In an implementation, theHDR level is based on a sum of the number of dark pixels and the numberof bright pixels. In an implementation, the HDR level is based on adifference between a sum of the number of dark pixels and the number ofbright pixels, and a product of the number of dark pixels and the numberof bright pixels. In an implementation, the processor is furtherconfigured to determine a value type for the HDR level, for the motionlevel, and for the scene luminance level, and apply fuzzy inferencerules to the value types to select the camera mode. In animplementation, the fuzzy inference rules are a three-dimensionaldecision cube with a motion axis, a HDR axis, and a light intensityaxis. In an implementation, the processor is further configured to applyspatial information to the number of dark pixels and the number ofbright pixels to determine the HDR level. In an implementation, thelight intensity inputs are exposure values. The second aspect mayinclude any combination of the features described in this paragraph andin the paragraphs of the first aspect and the third aspect.

A third aspect of this disclosure is a method for automated camera modeselection. The method includes determining high dynamic range (HDR)presence in a scene of an image to capture, detecting motion presencewithin the scene, determining scene luminance for the scene,automatically selecting a camera mode based on outputs from the HDRpresence determination, motion presence detection, and scene luminancedetermination, and capturing the image using the selected camera mode.In an implementation, the method further includes performing thedetermining the HDR presence, performing the detecting the motionpresence, and performing the determining scene luminance continuouslyuntil the image is captured. In an implementation, the method furtherincludes applying temporal smoothing to the outputs and applying medianfiltering to the selected camera mode. The third aspect may include anycombination of the features described in this paragraph and in theparagraphs of the first aspect and the second aspect.

These and other aspects of the present disclosure are disclosed in thefollowing detailed description, the appended claims, and theaccompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed implementations have other advantages and features thatwill be more readily apparent from the detailed description, theappended claims, and the accompanying figures. A brief introduction ofthe figures is below.

FIGS. 1A-D are isometric views of an example of an image capture device.

FIG. 2 is a cross-sectional view of an example of an image capturedevice including overlapping fields-of-view.

FIGS. 3A-B are block diagrams of examples of systems configured forimage capture.

FIG. 4 is a block diagram of an example of a camera mode selection andcapture pipeline.

FIG. 5 is a block diagram of an example of a parameter calculation unitof an automated camera mode selection unit of an image capture device.

FIGS. 6-11 are block diagrams of examples of pipelines for differentautomated camera mode selection techniques according to this disclosure.

FIG. 12 is a flowchart showing an example of a technique for automatedcamera mode selection.

DETAILED DESCRIPTION

An image capture device may capture the image according to image captureconfigurations of a selected camera mode. For example, a user of theimage capture device may select a camera mode to use to capture theimage. The selection may be based on one or more criteria including, forexample, an amount of background light, a location of a subject relativeto the image sensor, or a motion of the subject. Examples of cameramodes that may be available for selection include, without limitation, astill mode, still+local tone mapping (LTM) mode, high dynamic range(HDR) mode, and multi-frame noise reduction (MFNR) mode.

Each of those camera modes may be best suited for particular situations.For example, the still+LTM mode may be preferable where there is low tomid (e.g., 100 to 800) ISO, with or without motion, and a low amount ofnoise. In another example, the HDR mode may be preferable where there islow to mid ISO, the location of the image to capture is somewhereoutdoors, motion is detected up to a certain degree (small motion), andlow noise. In yet another example, the MFNR mode may be preferable wherethere is high (e.g., more than 800) ISO, up to a certain noise level,and without too much motion. The highest quality image or video mayresult from using the most preferable camera mode given the situation.

In many cases, the user of the image capture device may not select thebest camera mode to use at a given time or in a given place. Forexample, the user may not recognize that the background of an image tocapture does not have enough light for a selected mode or that an objectto be captured within the image has a motion that may not be capturedvery well using a selected mode. Furthermore, even if the user of theimage capture device selects an appropriate camera mode for capturing afirst image, that camera mode may not be the best camera mode forcapturing a subsequent image. That is, lighting, motion, or otherconditions within a location in which the images are captured may changeover a short amount of time. If the user does not account for thesechanges and select a new camera mode, the subsequently-captured imagemay be low quality.

Implementations of this disclosure address problems such as these usingautomated camera mode selection systems and techniques. Theimplementations of this disclosure are described in detail withreference to the drawings, which are provided as examples so as toenable those skilled in the art to practice the technology. The figuresand examples are not meant to limit the scope of the present disclosureto a single implementation or embodiment, and other implementations andembodiments are possible by way of interchange of, or combination with,some or all of the described or illustrated elements. Whereverconvenient, the same reference numbers will be used throughout thedrawings to refer to same or like parts.

FIGS. 1A-D are isometric views of an example of an image capture device100. The image capture device 100 may include a body 102 having a lens104 structured on a front surface of the body 102, various indicators onthe front of the surface of the body 102 (such as LEDs, displays, andthe like), various input mechanisms (such as buttons, switches, andtouch-screen mechanisms), and electronics (e.g., imaging electronics,power electronics, etc.) internal to the body 102 for capturing imagesvia the lens 104 and/or performing other functions. The image capturedevice 100 may be configured to capture images and video, and to storecaptured images and video for subsequent display or playback.

The image capture device 100 can include various indicators, includingthe LED lights 106 and the LED display 108. The image capture device 100can also include buttons 110 configured to allow a user of the imagecapture device 100 to interact with the image capture device 100, toturn the image capture device 100 on, and to otherwise configure theoperating mode of the image capture device 100. The image capture device100 can also include a microphone 112 configured to receive and recordaudio signals in conjunction with recording video. The side of the imagecapture device 100 may include an I/O interface 114. The camera may alsoinclude a microphone 116 system integrated into the camera housing. Thefront surface of the camera may include two drainage ports as part of adrainage channel 118 for the camera audio system. The camera can includean interactive display 120 that allows for interaction with the camerawhile simultaneously displaying camera information on a surface of thecamera. As illustrated, the image capture device 100 may include a lens104 configured to receive light incident upon the lens and to directreceived light onto an image sensor internal to the lens.

The image capture device 100 includes a camera exterior that encompassesand protects the camera's internal electronics, which are furtherdescribed in later sections. The camera exterior includes six surfaces(i.e. a front face, a left face, a right face, a back face, a top face,and a bottom face), wherein the exterior surfaces form a rectangularcuboid. Furthermore, both the front and rear surfaces of the imagecapture device 100 are substantially rectangular in shape. The imagecapture device 100 can be made of a rigid material such as plastic,aluminum, steel, or fiberglass. Additional camera features, such as thefeatures described above, may be affixed to an exterior of the camera.In some embodiments, the camera described herein includes features otherthan those described below. For example, instead of a single interfacebutton, the camera can include additional buttons or different interfacefeatures, such as a multiple microphone openings to receive voice orother audio commands.

Although not expressly shown in FIGS. 1A-D, in some implementations, theimage capture devices 100 may include one or more image sensors, such asa charge-coupled device (CCD) sensor, an active pixel sensor (APS), acomplementary metal-oxide semiconductor (CMOS) sensor, an N-typemetal-oxide-semiconductor (NMOS) sensor, and/or any other image sensoror combination of image sensors.

Although not expressly shown in FIGS. 1A-D, in some implementations, theimage capture device 100 may include one or more microphones, which mayreceive, capture, and record audio information, which may be associatedwith images acquired by the image sensors.

Although not expressly shown in FIGS. 1A-D, the image capture device 100may include one or more other information sources or sensors, such as aninertial measurement unit (IMU), a global positioning system (GPS)receiver component, a pressure sensor, a temperature sensor, a heartrate sensor, or any other unit, or combination of units, that may beincluded in an image capture apparatus.

In some implementations, the image capture device 100 may interface withor communicate with an external device, such as an external userinterface device, via a wired or wireless computing communication link(not shown). The user interface device may, for example, be the personalcomputing device 360 described below with respect to FIG. 3. Any numberof computing communication links may be used. The computingcommunication link may be a direct computing communication link or anindirect computing communication link, such as a link including anotherdevice or a network, such as the internet, may be used. In someimplementations, the computing communication link may be a Wi-Fi link,an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBee link,a near field communications (NFC) link, such as an ISO/IEC 23243protocol link, an Advanced Network Technology interoperability (ANT+)link, and/or any other wireless communications link or combination oflinks. In some implementations, the computing communication link may bean HDMI link, a USB link, a digital video interface link, a display portinterface link, such as a Video Electronics Standards Association (VESA)digital display interface link, an Ethernet link, a Thunderbolt link,and/or other wired computing communication link.

In some implementations, the image capture device 100 may transmitimages, such as panoramic images, or portions thereof, to the userinterface device (not shown) via the computing communication link, andthe user interface device may store, process, display, or a combinationthereof the panoramic images.

In some implementations, the user interface device may be a computingdevice, such as a smartphone, a tablet computer, a phablet, a smartwatch, a portable computer, and/or another device or combination ofdevices configured to receive user input, communicate information withthe image capture device 100 via the computing communication link, orreceive user input and communicate information with the image capturedevice 100 via the computing communication link.

In some implementations, the user interface device may display, orotherwise present, content, such as images or video, acquired by theimage capture device 100. For example, a display of the user interfacedevice may be a viewport into the three-dimensional space represented bythe panoramic images or video captured or created by the image capturedevice 100.

In some implementations, the user interface device may communicateinformation, such as metadata, to the image capture device 100. Forexample, the user interface device may send orientation information ofthe user interface device with respect to a defined coordinate system tothe image capture device 100, such that the image capture device 100 maydetermine an orientation of the user interface device relative to theimage capture device 100. Based on the determined orientation, the imagecapture device 100 may identify a portion of the panoramic images orvideo captured by the image capture device 100 for the image capturedevice 100 to send to the user interface device for presentation as theviewport. In some implementations, based on the determined orientation,the image capture device 100 may determine the location of the userinterface device and/or the dimensions for viewing of a portion of thepanoramic images or video.

In some implementations, the user interface device may implement orexecute one or more applications to manage or control the image capturedevice 100. For example, the user interface device may include anapplication for controlling camera configuration, video acquisition,video display, or any other configurable or controllable aspect of theimage capture device 100.

In some implementations, the user interface device, such as via anapplication, may generate and share, such as via a cloud-based or socialmedia service, one or more images, or short video clips, such as inresponse to user input. In some implementations, the user interfacedevice, such as via an application, may remotely control the imagecapture device 100, such as in response to user input.

In some implementations, the user interface device, such as via anapplication, may display unprocessed or minimally processed images orvideo captured by the image capture device 100 contemporaneously withcapturing the images or video by the image capture device 100, such asfor shot framing, which may be referred to herein as a live preview, andwhich may be performed in response to user input. In someimplementations, the user interface device, such as via an application,may mark one or more key moments contemporaneously with capturing theimages or video by the image capture device 100, such as with a tag,such as in response to user input.

In some implementations, the user interface device, such as via anapplication, may display, or otherwise present, marks or tags associatedwith images or video, such as in response to user input. For example,marks may be presented in a camera roll application for location reviewand/or playback of video highlights.

In some implementations, the user interface device, such as via anapplication, may wirelessly control camera software, hardware, or both.For example, the user interface device may include a web-based graphicalinterface accessible by a user for selecting a live or previouslyrecorded video stream from the image capture device 100 for display onthe user interface device.

In some implementations, the user interface device may receiveinformation indicating a user setting, such as an image resolutionsetting (e.g., 3840 pixels by 2160 pixels), a frame rate setting (e.g.,60 frames per second (fps)), a location setting, and/or a contextsetting, which may indicate an activity, such as mountain biking, inresponse to user input, and may communicate the settings, or relatedinformation, to the image capture device 100.

FIG. 2 is a cross-sectional view of an example of a dual-lens imagecapture device 200 including overlapping fields-of-view 210, 212. Insome implementations, the image capture device 200 may be a sphericalimage capture device with fields-of-view 210, 212 as shown in FIG. 2.For example, the image capture device 200 may include image capturedevices 220, 222, related components, or a combination thereof, arrangedin a back-to-back or Janus configuration. For example, a first imagecapture device 220 may include a first lens 230 and a first image sensor240, and a second image capture device 222 may include a second lens 232and a second image sensor 242 arranged oppositely from the first lens230 and the first image sensor 240.

The first lens 230 of the image capture device 200 may have thefield-of-view 210 shown above a boundary 250. Behind the first lens 230,the first image sensor 240 may capture a first hyper-hemispherical imageplane from light entering the first lens 230, corresponding to the firstfield-of-view 210.

The second lens 232 of the image capture device 200 may have afield-of-view 212 as shown below a boundary 252. Behind the second lens232, the second image sensor 242 may capture a secondhyper-hemispherical image plane from light entering the second lens 232,corresponding to the second field-of-view 212.

One or more areas, such as blind spots 260, 262, may be outside of thefields-of-view 210, 212 of the lenses 230, 232, light may be obscuredfrom the lenses 230, 232 and the corresponding image sensors 240, 242,and content in the blind spots 260, 262 may be omitted from capture. Insome implementations, the image capture device 200 may be configured tominimize the blind spots 260, 262.

The fields-of-view 210, 212 may overlap. Stitch points 270, 272,proximal to the image capture device 200, at which the fields-of-view210, 212 overlap may be referred to herein as overlap points or stitchpoints. Content captured by the respective lenses 230, 232, distal tothe stitch points 270, 272, may overlap.

Images contemporaneously captured by the respective image sensors 240,242 may be combined to form a combined image. Combining the respectiveimages may include correlating the overlapping regions captured by therespective image sensors 240, 242, aligning the captured fields-of-view210, 212, and stitching the images together to form a cohesive combinedimage.

A small change in the alignment, such as position and/or tilt, of thelenses 230, 232, the image sensors 240, 242, or both may change therelative positions of their respective fields-of-view 210, 212 and thelocations of the stitch points 270, 272. A change in alignment mayaffect the size of the blind spots 260, 262, which may include changingthe size of the blind spots 260, 262 unequally.

Incomplete or inaccurate information indicating the alignment of theimage capture devices 220, 222, such as the locations of the stitchpoints 270, 272, may decrease the accuracy, efficiency, or both ofgenerating a combined image. In some implementations, the image capturedevice 200 may maintain information indicating the location andorientation of the lenses 230, 232 and the image sensors 240, 242 suchthat the fields-of-view 210, 212, stitch points 270, 272, or both may beaccurately determined, which may improve the accuracy, efficiency, orboth of generating a combined image.

Optical axes through the lenses 230, 232 may be substantiallyantiparallel to each other, such that the respective axes may be withina tolerance such as 1%, 3%, 5%, 10%, and/or other tolerances. In someimplementations, the image sensors 240, 242 may be substantiallyperpendicular to the optical axes through their respective lenses 230,232, such that the image sensors may be perpendicular to the respectiveaxes to within a tolerance such as 1%, 3%, 5%, 10%, and/or othertolerances.

The lenses 230, 232 may be laterally offset from each other, may beoff-center from a central axis of the image capture device 200, or maybe laterally offset and off-center from the central axis. As compared toan image capture device with back-to-back lenses, such as lenses alignedalong the same axis, the image capture device 200 including laterallyoffset lenses 230, 232 may include substantially reduced thicknessrelative to the lengths of the lens barrels securing the lenses 230,232. For example, the overall thickness of the image capture device 200may be close to the length of a single lens barrel as opposed to twicethe length of a single lens barrel as in a back-to-back configuration.Reducing the lateral distance between the lenses 230, 232 may improvethe overlap in the fields-of-view 210, 212.

Images or frames captured by an image capture device, such as the imagecapture device 100 shown in FIGS. 1A-D or the image capture device 200shown in FIG. 2, may be combined, merged, or stitched together, toproduce a combined image, such as a spherical or panoramic image, whichmay be an equirectangular planar image. In some implementations,generating a combined image may include three-dimensional, orspatiotemporal, noise reduction (3DNR). In some implementations, pixelsalong the stitch boundary may be matched accurately to minimize boundarydiscontinuities.

FIGS. 3A-B are block diagrams of examples of systems configured forimage capture. Referring first to FIG. 3A, an image capture device 300configured for image capture is shown. The image capture device 300includes an image capture device 310 (e.g., a camera or a drone), whichmay, for example, be the image capture device 100 shown in FIGS. 1A-D.The image capture device 310 includes a processing apparatus 312 that isconfigured to receive a first image from the first image sensor 314 andreceive a second image from the second image sensor 316. The processingapparatus 312 may be configured to perform image signal processing(e.g., filtering, tone mapping, stitching, and/or encoding) to generateoutput images based on image data from the image sensors 314 and 316.

The image capture device 310 includes a communications interface 318 fortransferring images to other devices. The image capture device 310includes a user interface 320, which may allow a user to control imagecapture functions and/or view images. The image capture device 310includes a battery 322 for powering the image capture device 310. Thecomponents of the image capture device 310 may communicate with eachother via the bus 324.

The image capture device 300 may implement some or all of the pipelinesfor automated camera mode selection described in this disclosure, suchas the pipeline 400 of FIG. 4, the pipeline 600 of FIG. 6, the pipeline700 of FIG. 7, the pipeline 800 of FIG. 8, the pipeline 900 of FIG. 9,the pipeline 1000 of FIG. 10, the pipeline 1100 of FIG. 11, or acombination thereof. The image capture device 300 may be used toimplement some or all of the techniques described in this disclosure,such as the technique 1200 of FIG. 12.

The processing apparatus 312 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 312 mayinclude memory, such as a random-access memory device (RAM), flashmemory, or another suitable type of storage device such as anon-transitory computer-readable memory. The memory of the processingapparatus 312 may include executable instructions and data that can beaccessed by one or more processors of the processing apparatus 312. Forexample, the processing apparatus 312 may include one or more dynamicrandom access memory (DRAM) modules, such as double data ratesynchronous dynamic random-access memory (DDR SDRAM). In someimplementations, the processing apparatus 312 may include a digitalsignal processor (DSP). In some implementations, the processingapparatus 312 may include an application specific integrated circuit(ASIC). For example, the processing apparatus 312 may include a customimage signal processor.

The first image sensor 314 and the second image sensor 316 areconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 314 and 316 may include CCDs or active pixelsensors in a CMOS. The image sensors 314 and 316 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensors 314 and 316 include digital-to-analogconverters. In some implementations, the image sensors 314 and 316 areheld in a fixed orientation with respective fields of view that overlap.

The image capture device 310 may include a communications interface 318,which may enable communications with a personal computing device (e.g.,a smartphone, a tablet, a laptop computer, or a desktop computer). Forexample, the communications interface 318 may be used to receivecommands controlling image capture and processing in the image capturedevice 310. For example, the communications interface 318 may be used totransfer image data to a personal computing device. For example, thecommunications interface 318 may include a wired interface, such as ahigh-definition multimedia interface (HDMI), a universal serial bus(USB) interface, or a FireWire interface. For example, thecommunications interface 318 may include a wireless interface, such as aBluetooth interface, a ZigBee interface, and/or a Wi-Fi interface.

The image capture device 310 may include a user interface 320. Forexample, the user interface 320 may include an LCD display forpresenting images and/or messages to a user. For example, the userinterface 320 may include a button or switch enabling a person tomanually turn the image capture device 310 on and off. For example, theuser interface 320 may include a shutter button for snapping pictures.The image capture device 310 may include a battery 322 that powers theimage capture device 310 and/or its peripherals. For example, thebattery 322 may be charged wirelessly or through a micro-USB interface.

In some implementations, the image capture device 310 may include one ormore hardware or software components for performing global tone mappingagainst pixels of an image captured using the image capture device 310.The global tone mapping performed using those one or more hardware orsoftware components may integrate color correction operations. Forexample, those one or more hardware or software components may be usedto perform the technique 1200 described below with respect to FIG. 12.

Referring next to FIG. 3B, a system 330 configured for image capture isshown. The system 330 includes an image capture device 340 and apersonal computing device 360 that communicate via a communications link350. The image capture device 340 may, for example, be the image capturedevice 100 shown in FIGS. 1A-D. The personal computing device 360 may,for example, be the user interface device described with respect toFIGS. 1A-D. The image capture device 340 includes a first image sensor342 and a second image sensor 344 that are configured to capturerespective images. The image capture device 340 includes acommunications interface 346 configured to transfer images via thecommunication link 350 to the personal computing device 360.

The personal computing device 360 includes a processing apparatus 362that is configured to receive, using the communications interface 366, afirst image from the first image sensor, and receive a second image fromthe second image sensor 344. The processing apparatus 362 may beconfigured to perform image signal processing (e.g., filtering, tonemapping, stitching, and/or encoding) to generate output images based onimage data from the image sensors 342 and 344.

The image capture device 340 may implement some or all of the pipelinesfor automated camera mode selection described in this disclosure, suchas the pipeline 400 of FIG. 4, the pipeline 600 of FIG. 6, the pipeline700 of FIG. 7, the pipeline 800 of FIG. 8, the pipeline 900 of FIG. 9,the pipeline 1000 of FIG. 10, the pipeline 1100 of FIG. 11, or acombination thereof. The image capture device 340 may be used toimplement some or all of the techniques described in this disclosure,such as the technique 1200 of FIG. 12.

The first image sensor 342 and the second image sensor 344 areconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 342 and 344 may include CCDs or active pixelsensors in a CMOS. The image sensors 342 and 344 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensors 342 and 344 include digital-to-analogconverters. In some implementations, the image sensors 342 and 344 areheld in a fixed relative orientation with respective fields of view thatoverlap. Image signals from the image sensors 342 and 344 may be passedto other components of the image capture device 340 via the bus 348.

The communications link 350 may be a wired communications link or awireless communications link. The communications interface 346 and thecommunications interface 366 may enable communications over thecommunications link 350. For example, the communications interface 346and the communications interface 366 may include an HDMI port or otherinterface, a USB port or other interface, a FireWire interface, aBluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. Forexample, the communications interface 346 and the communicationsinterface 366 may be used to transfer image data from the image capturedevice 340 to the personal computing device 360 for image signalprocessing (e.g., filtering, tone mapping, stitching, and/or encoding)to generate output images based on image data from the image sensors 342and 344.

The processing apparatus 362 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 362 mayinclude memory, such as RAM, flash memory, or another suitable type ofstorage device such as a non-transitory computer-readable memory. Thememory of the processing apparatus 362 may include executableinstructions and data that can be accessed by one or more processors ofthe processing apparatus 362. For example, the processing apparatus 362may include one or more DRAM modules, such as DDR SDRAM.

In some implementations, the processing apparatus 362 may include a DSP.In some implementations, the processing apparatus 362 may include anintegrated circuit, for example, an ASIC. For example, the processingapparatus 362 may include a custom image signal processor. Theprocessing apparatus 362 may exchange data (e.g., image data) with othercomponents of the personal computing device 360 via the bus 368.

The personal computing device 360 may include a user interface 364. Forexample, the user interface 364 may include a touchscreen display forpresenting images and/or messages to a user and receiving commands froma user. For example, the user interface 364 may include a button orswitch enabling a person to manually turn the personal computing device360 on and off. In some implementations, commands (e.g., start recordingvideo, stop recording video, or snap photograph) received via the userinterface 364 may be passed on to the image capture device 340 via thecommunications link 350.

In some implementations, the image capture device 340 and/or thepersonal computing device 360 may include one or more hardware orsoftware components for performing global tone mapping against pixels ofan image captured using the image capture device 340. The global tonemapping performed using those one or more hardware or softwarecomponents may integrate color correction operations. For example, thoseone or more hardware or software components may be used to perform thetechnique 1200 described below with respect to FIG. 12.

FIG. 4 is a block diagram of an example of a camera mode selection andcapture pipeline 400. In some implementations, the camera mode selectionand capture pipeline 400 may be included in an image capture device,such as the image capture device 100 shown in FIGS. 1A-D or the imagecapture device 200 shown in FIG. 2. In some implementations, the cameramode selection and capture pipeline 400 may represent functionality ofan integrated circuit, for example, including an image capture unit, acamera mode selection unit, or a combined camera mode selection andimage capture unit.

The camera mode selection and capture pipeline 400 receives input 402and processes the input 402 to produce output 404. The input 402 may beinformation or measurements usable to select a camera mode at anautomated camera mode selection unit 406. For example, the input 402 mayinclude measurements related to criteria processed by the automatedcamera mode selection unit 406, such as dynamic range, motion, and/orlight intensity. The input 402 may be received using one or more sensorsof the image capture device or processor implementing the camera modeselection and capture pipeline 400.

The output 404 may be an image captured using an image capture unit 408.The image capture unit 408 uses the camera mode selected by theautomated camera mode selection unit 406 to capture an image, such asusing an image sensor (e.g., the first image sensor 314 and/or thesecond image sensor 316, or the first image sensor 342 and/or the secondimage sensor 344). For example, the image captured using the imagecapture unit 408 may be an image or a frame of a video. That image orframe may be one of a sequence or series of images or frames of a video,such as a sequence, or series, of frames captured at a rate, or framerate, which may be a number or cardinality of frames captured perdefined temporal period, such as twenty-four, thirty, or sixty framesper second. The output 404 may be output for display at the imagecapture device and/or transmitted to another component or device.

The automated camera mode selection unit 406 includes a parametercalculation unit 410 and a mode selection unit 412. The parametercalculation unit 410 processes the input 402 to determine values for theimage selection criteria represented within the input 402. The parametercalculation unit 410 outputs those values or data indicative thereof tothe mode selection unit 412. The mode selection unit 412 selects acamera mode to use to capture an image based on those values or data. Insome implementations, the mode selection unit 412 may select the cameramode based at least in part on secondary input 414.

In some implementations, the automated camera mode selection unit 406may include additional units or functionality. In some implementations,the parameter calculation unit 410 and the mode selection unit 412 maybe combined into one unit. In some implementations, aspects of one orboth of the parameter calculation unit 410 or the mode selection unit412 may be separated into multiple units.

FIG. 5 is a block diagram of an example of a parameter calculation unit500 of an automated camera mode selection unit of an image capturedevice. For example, the parameter calculation unit 500 may be theparameter calculation unit 410 of the pipeline 400 described withrespect to FIG. 4. The parameter calculation unit 500 includes a HDRestimation unit 502, a motion estimation unit 504, and a light intensityestimation unit 506. The HDR estimation unit 502 estimates dynamic rangefor an image to be captured using first input 508. The motion estimationunit 504 estimates motion for the image to be captured using secondinput 510. The light intensity estimation unit 506 estimates lightintensity for the image to be captured using third input 512.

Some or all of the first input 508, the second input 510, or the thirdinput 512 may be included within input received at the automated cameramode selection unit that implements the parameter calculation unit 500.For example, some or all of the first input 508, the second input 510,or the third input 512 may be received within the input 402 describedwith respect to FIG. 4. In some implementations, some or all of thefirst input 508, the second input 510, or the third input 512 may referto the same or similar information or measurements.

The HDR estimation unit 502 uses the first input 508 to determinewhether a scene for the image to be captured is HDR or non-HDR. The HDRestimation unit 502 determines that the scene for the image to becaptured is non-HDR if the first input 508 indicates that the imagesensor of the image capture device will be able to capture the entireinformation of the scene without resulting in dark or saturated pixels.Similarly, the HDR estimation unit 502 determines that the scene for theimage to be captured is HDR if there are at least a threshold number ofdark pixels and a threshold number of bright pixels. Those thresholdnumbers may be the same or different. As such, the HDR estimation unit502 processes the first input 508 to determine a number of dark pixelsand a number of bright pixels.

The motion estimation unit 504 uses the second input 510 to determinewhether camera motion is detected. For example, the motion estimationunit 504 can determine whether sensor measurements (e.g., angular speed)indicated within the second input 510 meets a motion threshold. If themotion threshold is met (e.g., because the angular speed is higher thana value for the motion threshold), the motion estimation unit 504determines that camera motion is detected.

The light intensity estimation unit 506 uses the third input 512 todetermine a light intensity for the scene for the image to be captured.Thus, the light intensity may be estimated based on data representingthe scene luminance within the third input 512. As such, the lightintensity estimation unit 506 may determine the light intensity for thescene based on a light intensity threshold. For example, the lightintensity estimation unit 506 may determine that the light intensity islow where the scene luminance is below the light intensity threshold. Inanother example, the light intensity estimation unit 506 may determinethat the light intensity is high where the scene luminance is higherthan the light intensity threshold. In yet another example, the lightintensity estimation unit 506 may determine that the light intensity ismedium where the scene luminance is neither lower nor higher than thelight intensity threshold.

Implementations and examples of automated camera mode selectionaccording to this disclosure may use different inputs for the parametercalculations described with respect to FIGS. 4 and 5 and/or performdifferent processing than as described with respect to FIGS. 4 and 5. Inparticular, FIGS. 6-11 are block diagrams of examples of pipelines fordifferent automated camera mode selection techniques according to thisdisclosure.

Referring first to FIG. 6, a pipeline 600 used to implement a firsttechnique for automated camera mode selection is shown. According to thefirst technique, a HDR estimation unit 602 (e.g., the HDR estimationunit 502 described with respect to FIG. 5) receives two input values 604and 606 (e.g., the first input 508 described with respect to FIG. 5).The two input values 604 and 606 refer to control points defined usingone or more look up tables (LUTs). Each of the two input values 604 and606 is compared to a threshold. In some cases, both of the two inputvalues 604 and 606 may be compared to the same threshold.

In particular, the two input values 604 and 606 represent control pointsused to define what a dark pixel is and what a bright pixel is. A firstLUT corresponding to the first input value 604 corresponds to a firstcurve for the image and a second LUT corresponding to the second inputvalue 606 corresponds to a second curve for the image. The first curveof the first LUT and the second curve of the second LUT are weightingcurves. The weighting curves are exponential functions between 0.0 and1.0. These curves are vectors (the LUTs) of 256 values. On the otherhand, the 64×48 thumbnail is represented using 16 bits. Each pixel valueof the thumbnail is transformed to an 8 bits value. This 8 bits value isthe index for the LUTs, i.e., Score_bright=LUT_bright[8bits_value] andScore_dark=LUT_dark[8bits_value], for example. That is, each curve (anexponential function) is parametrized by one control point (tuningparameter). The transformation to 8 bits is done for all pixels of thethumbnail and all the scores are summed. These scores are thennormalized to get F_(Dark) and F_(Bright).

Thus, the HDR estimation unit 602 determines whether a scene for theimage to be captured is HDR based on where the two input values 604 and606 are with respect to the weighting curves. The HDR estimation unit602 outputs an indicator 608 as a result of that determination. Theindicator 608 is a Boolean in which a first value (e.g., true) indicatesthat the HDR estimation unit 602 determined the scene to be HDR and inwhich a second value (e.g., false) indicates that the HDR estimationunit 602 determined the scene to be non-HDR.

The motion estimation unit 610 (e.g., the motion estimation unit 504described with respect to FIG. 5) receives one input value 612 (e.g.,the second input 510 described with respect to FIG. 5) representing anangular speed of an object detected using a gyroscope or other sensor.The motion estimation unit 610 determines whether the input value 612 ishigher than a first threshold. If the input value 612 is higher than thefirst threshold, the motion estimation unit 610 determines that cameramotion is detected.

If the input value 612 is not higher than the first threshold, a Sum ofAbsolute Differences (SAD) is determined between the current and theprevious thumbnails. If the SAD is higher than a second threshold, themotion estimation unit 610 determines that scene motion is detected. Ifthe SAD is not higher than the second threshold, the motion estimationunit 610 determines that no motion is detected. The motion estimationunit 610 outputs an indicator 614 as a result of the one or twodeterminations. The indicator 614 is a Boolean in which a first value(e.g., true) indicates that the motion estimation unit 610 detected somemotion and in which a second value (e.g., false) indicates that themotion estimation unit 610 did not detect motion.

The light intensity estimation unit 616 (e.g., the light intensityestimation unit 506 described with respect to FIG. 5) receives one inputvalue 618 (e.g., the third input 512 described with respect to FIG. 5)representing a scene luminance estimated within exposure correctionoperations. The input value 618 is compared against a threshold forlight intensity. If the input value 618 is greater than the threshold,the light intensity estimation unit 616 determines that the lightintensity is high. If the input value 618 is less than the threshold,the light intensity estimation unit 616 determines that the lightintensity is low. If the input value 618 is neither greater nor lessthan the threshold, the light intensity estimation unit 616 determinesthat the light intensity is medium.

The light intensity estimation unit 616 outputs an indicator 620 basedon the comparison between the input value 618 and the threshold. Forexample, the indicator can have a first value when the light intensityis determined to be high, a second value when the light intensity isdetermined to be low, and a third value when the light intensity isdetermined to be medium.

A mode selection unit 622 (e.g., the mode selection unit 412 describedwith respect to FIG. 4) selects a camera mode to use to capture an imagebased on the indicator 608, the indicator 614, and the indicator 620.For example, each of the camera modes that may be selected may bedefined to correspond to particular combinations of values of theindicator 608, the indicator 614, and the indicator 620. As such, theparticular combination of values of the indicator 608, the indicator614, and the indicator 620 may be used to identify one of the cameramodes. As a result, a selected camera mode 624 is selected. The selectedcamera mode 624 may then be used, such as by an image capture unit(e.g., the image capture unit 408 described with respect to FIG. 4), tocapture an image.

Referring next to FIG. 7, a pipeline 700 used to implement a secondtechnique for automated camera mode selection is shown. The secondtechnique implemented using the pipeline 700 represents an extension ofthe first technique implemented using the pipeline 600 described withrespect to FIG. 6. The second technique addresses temporal instability(e.g., output flickering) that may result from using the thresholdvalues to produce the indicator 608 as output of the HDR estimation unit602 and to produce the indicator 614 as output of the motion estimationunit 610. In particular, the second technique introduces fuzzy logic toimprove the behavior and the robustness of the automated camera modeselection process, and further introduces temporal smoothing filteringin which inputs are smoothed.

The pipeline 700 includes an HDR estimation unit 702, a motionestimation unit 704, a light intensity estimation unit 706, and a modeselection unit 708 (which may, for example, be the HDR estimation unit502, the motion estimation unit 504, and the light intensity estimationunit 506 described with respect to FIG. 5, and the mode selection unit412 described with respect to FIG. 4). The output of each of the HDRestimation unit 702, the motion estimation unit 704, and the lightintensity estimation unit 706 is expressed as a fuzzy value between 0.0and 1.0. The mode selection unit 708 processes the fuzzy values outputfrom each of the HDR estimation unit 702, the motion estimation unit704, and the light intensity estimation unit 706. In particular, themode selection unit generally performs fuzzification against the fuzzyvalues received as input, evaluates the fuzzy values in view of rules,aggregates the output of those rules, and performs defuzzificationagainst the aggregated output.

According to the second technique, the HDR estimation unit 702 receivestwo input values 710 and 712 (e.g., the first input 508 described withrespect to FIG. 5). Processing is performed similar to that describedwith respect to the HDR estimation unit 602 of the pipeline 600described with respect to FIG. 6. In particular, the sum of dark pixels(F_(Dark)) is determined and the sum of bright pixels (F_(Bright)) isdetermined, using one or more dedicated weighting functions. The sumsF_(Dark) and F_(Bright) are then normalized to determine an output 714as a fuzzy value F_(HDR).

Where an AND condition is used for the HDR detection, such that whetherHDR is detected is based on the sum of both of the dark pixels and thebright pixels, the output 714 is calculated by F_(Dark) and F_(Bright).Alternatively, where an OR condition is used for the HDR detection, suchthat whether HDR is detected is based on the sum of the dark pixels orthe sum of the bright pixels, the output 714 is calculated as thedifference of a first value and a second value, where the first valuerepresents the sum of F_(Dark) and F_(Bright) and the second valuerepresents the product of F_(Dark) and F_(Bright). In someimplementations, instead of two sums, a single sum of the mid-tonepixels may instead be determined.

The motion estimation unit 704 receives one input value 716 (e.g., thesecond input 510 described with respect to FIG. 5) representing anangular speed (F_(Gyro)) of an object detected using a gyroscope orother sensor. In an implementation, F_(Gyro) is the normalized value(between 0 and 1) of the angular speed (input of the motion estimationunit). Processing is performed similar to that described with respect tothe motion estimation unit 610 of the pipeline 600. In particular, themotion estimation unit 704 determines whether the input value 716 ishigher than a threshold (T_(GyroSpeed)). If the input value 716 ishigher than T_(GyroSpeed), the motion estimation unit 704 determinesthat camera motion is detected.

However, if the input value 716 is not higher than T_(GyroSpeed), a SADis determined between two consecutive thumbnails (e.g., the current andthe previous thumbnails). The SAD value is then normalized to producenormalized value F_(Img). As such, if the input value 716 is lower thanT_(GyroSpeed), an output 718 of the motion estimation unit 704,F_(Motion), is expressed as a fuzzy value representing F_(Img).Otherwise, the output 718 is expressed as a fuzzy value representingF_(Gyro).

The light intensity estimation unit 706 receives one input value 720(e.g., the third input 512 described with respect to FIG. 5)representing a scene luminance estimated within exposure correctionoperations. Processing is performed similar to that described withrespect to the light intensity estimation unit 616 of the pipeline 600.In particular, the input value 720 is compared against a threshold forlight intensity. The input value 720 is then normalized to produce anoutput 722. The output 722 represents a fuzzy value ofF_(LightIntensity).

A mode selection unit 708 selects a camera mode to use to capture animage based on the fuzzy values included within the output 714, theoutput 718, and the output 722. First, the mode selection unit 708fuzzifies the output 714, the output 718, and the output 722. Forexample, fuzzifying the output 714, the output 718, and the output 722can include identifying each of the output 714, the output 718, and theoutput 722 as one of a small value, a medium value, or a large value(e.g., determining degrees of membership within those size categories).The mode selection unit 708 then evaluates the fuzzified values(expressed simply as “motion,” “dynamic range,” and “light intensity”)in view of fuzzy inference rules. In some implementations, the fuzzyinference rules may be represented as a three-dimensional decision cube.For example, each of the three axes of the three-dimensional decisioncube may represent one of motion, dynamic range, or light intensity.

Examples of the fuzzy inference rules include, without limitation: (1)if motion is small and dynamic range is small and light intensity issmall, then mode is MFNR; (2) if motion is small and dynamic range islarge and light intensity is small, then mode is MFNR; (3) if motion issmall and dynamic range is small and light intensity is medium, thenmode is MFNR; (4) if motion is small and dynamic range is large andlight intensity is medium, then mode is HDR; (5) if motion is small anddynamic range is small and light intensity is large, then mode isSTILL+LTM; (6) if motion is small and dynamic range is large and lightintensity is large, then mode is HDR; (7) if motion is large and dynamicrange is small and light intensity is small, then mode is STILL; (8) ifmotion is large and dynamic range is large and light intensity is small,then mode is STILL; (9) if motion is large and dynamic range is smalland light intensity is medium, then mode is STILL+LTM; (10) if motion islarge and dynamic range is large and light intensity is medium, thenmode is STILL+LTM; (11) if motion is large and dynamic range is smalland light intensity is large, then mode is STILL+LTM; and (12) if motionis large and dynamic range is large and light intensity is large, thenmode is STILL+LTM.

Evaluating the fuzzified values using the fuzzy inference rules includesdetermining scores for each of the fuzzified values. For example, asmall value of motion may have a score of X, where a large value ofmotion may have a score of Y. In another example, a small value ofdynamic range may have a score of M, where a large value of dynamicrange may have a score of N. In yet another example, a small value oflight intensity may have a score of A, where a medium value of lightintensity may have a score of B, and where a large value of lightintensity may have a score of C. The scores for each of the threefuzzified values are multiplied to determine a score for a given one ofthe fuzzy inference rules.

The fuzzy inference rule associated with the highest resulting score maybe selected. In some cases, there may be multiple fuzzy inference rulesthat correspond to a single camera mode. In such a case, the fuzzyinference rule having the highest score for that single camera mode isused instead of the other fuzzy inference rules. The mode selection unit708 may then select the camera mode corresponding to the selected fuzzyinference rule.

The mode selection unit 708 then defuzzifies the fuzzy values used forthe selected fuzzy inference rule. For example, defuzzifiying fuzzyvalues may include plotting a three-dimensional decision cube of thosefuzzy values for tuning. For example, the mode selection unit 708 mayinclude a temporal smoothing unit 724. The temporal smoothing unit 724processes the fuzzy values corresponding to the selected fuzzy inferencerule using temporal smoothing filtering, such as to avoid instabilities.For example, the temporal smoothing unit 724 can process given fuzzyvalues as F_(i,t)=alpha_(i)*F_(i,t-1)+(1−alpha_(i))*F_(i,t), where tmeans time or frame index and i means “Dark”, “Bright”, “Gyro”,“Spatial”, “Histo, and the like, for example. In some implementations,the temporal smoothing unit 724 may be external to the mode selectionunit 708.

As a result, a selected camera mode 726 is selected. The selected cameramode 726 may then be used, such as by an image capture unit (e.g., theimage capture unit 408 described with respect to FIG. 4), to capture animage.

Referring next to FIG. 8, a pipeline 800 used to implement a thirdtechnique for automated camera mode selection is shown. The thirdtechnique implemented using the pipeline 800 represents an extension ofthe second technique implemented using the pipeline 700 described withrespect to FIG. 7. The third technique introduces spatial analysis forimproving the output of the HDR estimation unit 702 and the use ofexposure values for improving the output of the light intensityestimation unit 706. In particular, the third technique uses spatialinformation in addition to or in place of bright and dark pixelinformation for HDR detection.

The pipeline 800 includes an HDR estimation unit 802, a motionestimation unit 804, a light intensity estimation unit 806, and a modeselection unit 808 (which may, for example, be the HDR estimation unit502, the motion estimation unit 504, and the light intensity estimationunit 506 described with respect to FIG. 5, and the mode selection unit412 described with respect to FIG. 4). The output of each of the HDRestimation unit 802, the motion estimation unit 804, and the lightintensity estimation unit 806 is expressed as a fuzzy value between 0.0and 1.0.

According to the third technique, the HDR estimation unit 802 receivestwo input values 810 and 812 (e.g., the first input 508 described withrespect to FIG. 5). Processing is performed similar to that describedwith respect to the HDR estimation unit 702 of the pipeline 700described with respect to FIG. 7. In particular, the sum of dark pixels(F_(Dark)) is determined and the sum of bright pixels (F_(Bright)) isdetermined, using one or more dedicated weighting functions. The sumsF_(Dark) and F_(Bright) are then normalized to determine an output 814as a fuzzy value F_(HDR).

However, whereas other techniques for automated camera mode selectionuse the sums F_(Dark) and F_(Bright) to determine whether HDR isdetected, the HDR estimation unit 802 further uses spatial information816 for the dark pixels and for the bright pixels to detect HDR. Forexample, the HDR estimation unit 802 operates under the principle that ascene with HDR should have a difference of intensity between the centerand the border of the scene (e.g., in backlight conditions). As such,the HDR estimation unit 802 uses the spatial information 816 to detectdifferences in the background and in the foreground of the scene. Thespatial information 816 may include, for example, a saliency map or asimilar mechanism.

The absolute difference between the average of the background andforeground regions of the scene can be normalized to detect HDR. Forexample, a value F_(Histo) can be defined as the product of F_(Dark) andF_(Bright). Using the spatial analysis introduced within the HDRestimation unit 802, the output 814 (F_(HDR)) can be determined based onwhether the spatial information 816 (e.g., backlight detection) is usedalong with or instead of the sums F_(Dark), and F_(Bright). For example,where the spatial information 816 is used along with the sums F_(Dark),and F_(Bright), the output 814 can be expressed as the product ofF_(Histo) and F_(Spatial). In an implementation, F_(Spatial) is thenormalized value (between 0 and 1) of a pattern difference value, wherea pattern is a small 3×3 matrix that is composed of black and whiteareas. This pattern is applied to the thumbnail to compute thedifference between the white areas and the black areas. In anotherexample, where the spatial information 816 is used instead of the sumsF_(Dark) and F_(Bright), the output 814 can be expressed as thedifference between a first value and a second value, where the firstvalue is the sum of F_(Histo) and F_(Spatial) and where the second valueis the product of F_(Histo) and F_(Spatial).

The motion estimation unit 804 receives one input value 818 (e.g., thesecond input 510 described with respect to FIG. 5). Processing isperformed similar to that described with respect to the motionestimation unit 704 of the pipeline 700 described with respect to FIG.7. In particular, the motion estimation unit 804 determines whether theinput value 818 is higher than the threshold T_(GyroSpeed). If the inputvalue 818 is higher than T_(GyroSpeed), the motion estimation unit 804determines that camera motion is detected. However, if the input value818 is not higher than T_(GyroSpeed), a SAD is determined between twoconsecutive thumbnails (e.g., the current and the previous thumbnails).The SAD value is then normalized to produce normalized value F_(Img). Assuch, if the input value 818 is lower than T_(GyroSpeed), an output 820of the motion estimation unit 804, F_(Motion), is expressed as a fuzzyvalue representing F_(Img). Otherwise, the output 820 is expressed as afuzzy value representing F_(Gyro).

The light intensity estimation unit 806 receives one input value 822(e.g., the third input 512 described with respect to FIG. 5). Processingis performed similar to that described with respect to the lightintensity estimation unit 706 of the pipeline 700 described with respectto FIG. 7. In particular, the input value 822 is compared against athreshold for light intensity. The input value 822 is then normalized toproduce an output 824. The output 824 represents a fuzzy value ofF_(LightIntensity).

The mode selection unit 808 receives the output 814, the output 820, andthe output 824. Processing is performed similar to that described withrespect to the mode selection unit 708 of the pipeline 700 describedwith respect to FIG. 7. In particular, the mode selection unit 808generally performs fuzzification against the output 814, the output 820,and the output 824, evaluates the fuzzy values thereof in view of rules,aggregates the output of those rules, and performs defuzzificationagainst the aggregated output (e.g., using a temporal smoothing unit826). In some implementations, the temporal smoothing unit 826 may beexternal to the mode selection unit 808. As a result, a selected cameramode 828 is selected. The selected camera mode 828 may then be used,such as by an image capture unit (e.g., the image capture unit 408described with respect to FIG. 4), to capture an image.

Referring next to FIG. 9, a pipeline 900 used to implement a fourthtechnique for automated camera mode selection is shown. The fourthtechnique implemented using the pipeline 900 represents an extension ofthe first technique implemented using the pipeline 600 described withrespect to FIG. 6. The fourth technique allows for the light intensityto be estimated using exposure values or camera ISO and further usescuboids defined by tuning for the mode selection, such as instead ofusing fuzzy values.

In particular, the fourth technique uses parameter tuning to define theareas of activation for each camera mode within a three-dimensionaldecision cube. Values for each of the camera modes correspond todefined, non-overlapping three-dimensional regions within thethree-dimensional decision cube. The three-dimensional decision cubemay, for example, be the three-dimensional decision cube produced andused in connection with the second technique and/or the third technique,respectfully described above with respect to FIGS. 7 and 8.

The pipeline 900 includes an HDR estimation unit 902, a motionestimation unit 904, a light intensity estimation unit 906, a modeselection unit 908 (which may, for example, be the HDR estimation unit502, the motion estimation unit 504, and the light intensity estimationunit 506 described with respect to FIG. 5, and the mode selection unit412 described with respect to FIG. 4), and a parameter tuning unit 910.

According to the fourth technique, the HDR estimation unit 902 receivestwo input values 912 and 914 (e.g., the first input 508 described withrespect to FIG. 5). Processing is performed similar to that describedwith respect to the HDR estimation unit 602 of the pipeline 600described with respect to FIG. 6. In particular, the HDR estimation unit902 determines whether a scene for the image to be captured is HDR basedon where the two input values 912 and 914 are with respect to theweighting curves. The HDR estimation unit 902 outputs an output 916 as aresult of that determination. The output 916 is a Boolean indicator inwhich a first value (e.g., true) indicates that the HDR estimation unit902 determined the scene to be HDR and in which a second value (e.g.,false) indicates that the HDR estimation unit 902 determined the sceneto be non-HDR.

The motion estimation unit 904 receives one input value 918 (e.g., thesecond input 510 described with respect to FIG. 5). Processing isperformed similar to that described with respect to the motionestimation unit 610 of the pipeline 600 described with respect to FIG.6. In particular, the motion estimation unit 904 determines whether theinput value 918 is higher than a first threshold. If the input value 918is higher than the first threshold, the motion estimation unit 904determines that camera motion is detected. If the input value 918 is nothigher than the first threshold, a SAD is determined between the currentand the previous thumbnails. If the SAD is higher than a secondthreshold, the motion estimation unit 904 determines that scene motionis detected. If the SAD is not higher than the second threshold, themotion estimation unit 904 determines that no motion is detected. Themotion estimation unit 904 outputs an output 920, a Boolean indicator,as a result of the one or two determinations.

The light intensity estimation unit 906 receives one input value 922(e.g., the third input 512 described with respect to FIG. 5). Processingis performed similar to that described with respect to the lightintensity estimation unit 616 of the pipeline 600 described with respectto FIG. 6. In particular, the input value 922 represents a sceneluminance estimated within exposure correction operations. The inputvalue 922 is compared against a threshold for light intensity. If theinput value 922 is greater than the threshold, the light intensityestimation unit 906 determines that the light intensity is high. If theinput value 922 is less than the threshold, the light intensityestimation unit 906 determines that the light intensity is low. If theinput value 922 is neither greater nor less than the threshold, thelight intensity estimation unit 906 determines that the light intensityis medium.

The light intensity estimation unit 906 produces an output 924 toindicate the results of the comparison between the input value 922 andthe threshold. However, whereas the input value for the light intensityestimation unit 616 of the pipeline 600 is an exposure value, the inputvalue 922 may be an exposure value or an ISO value. As such, the lightintensity estimation unit 906 may produce the output 924 based oncomparisons between the described thresholds and the ISO value receivedas the input value 922.

The output 916, the output 920, and the output 924 may each berepresented as a set of three values, each between 0.0 and 1.0. Thosevalues correspond to a location within a three-dimensional region of thethree-dimensional decision cube. The parameter tuning unit 910 receivesthe output 916, the output 920, and the output 924 and determines, basedon the values included in each of those, the three-dimensional region ofthe three-dimensional decision cube to which the output 916, the output920, and the output 924 correspond. Data indicative of thatthree-dimensional region is then passed to the mode selection unit 908,which selects the selected camera mode 926 as the camera mode thatcorresponds to that three-dimensional region. The selected camera mode926 may then be used, such as by an image capture unit (e.g., the imagecapture unit 408 described with respect to FIG. 4), to capture an image.

The parameter tuning unit 910 is also used to update thethree-dimensional decision cube used by the mode selection unit 908 toselect the camera mode. That is, over time, the three-dimensionalregions of the three-dimensional decision cube may change in size orposition, such as based on the inputs received by the pipeline 900(e.g., the input 402 described with respect to FIG. 4). The parametertuning unit 910 processes these changes to update the three-dimensionaldecision cube accordingly.

In some implementations, the parameter tuning unit 910 may not belocated before the mode selection unit 908 in the pipeline 900. Forexample, the mode selection unit 908 may directly receive the output916, the output 920, and the output 924. The mode selection unit 908 maythen use the parameter tuning unit 910 to identify the selected cameramode 926. For example, the mode selection unit 908 can send thethree-dimensional index values of the output 916, the output 920, andthe output 924 to the parameter tuning unit 910. The parameter tuningunit 910 may then to query the three-dimensional decision cube for theselected camera mode 926 according to those three-dimensional indexvalues.

Referring next to FIG. 10, a pipeline 1000 used to implement a fifthtechnique for automated camera mode selection is shown. The fifthtechnique implemented using the pipeline 1000 represents an extension ofthe fourth technique implemented using the pipeline 900 described withrespect to FIG. 9. The fifth technique improves the temporal smoothingfiltering functionality of the automated camera mode selection processby further smoothing the output of the temporal smoothing. Inparticular, the fifth technique uses temporal smoothing filtering inaddition to parameter tuning to select a camera mode based on outputfrom processing units of the pipeline 1000.

The pipeline 1000 includes an HDR estimation unit 1002, a motionestimation unit 1004, a light intensity estimation unit 1006, a modeselection unit 1008 (which may, for example, be the HDR estimation unit502, the motion estimation unit 504, and the light intensity estimationunit 506 described with respect to FIG. 5, and the mode selection unit412 described with respect to FIG. 4), and a parameter tuning unit 1010(which may, for example, be the parameter tuning unit 910 described withrespect to FIG. 9).

According to the fifth technique, the HDR estimation unit 1002 receivestwo input values 1010 and 1012 (e.g., the first input 508 described withrespect to FIG. 5). Processing is performed similar to that describedwith respect to the HDR estimation unit 602 of the pipeline 600described with respect to FIG. 6. In particular, the HDR estimation unit1002 determines whether a scene for the image to be captured is HDRbased on where the two input values 1012 and 1014 are with respect tothe weighting curves. The HDR estimation unit 1002 outputs an output1016 as a result of that determination. The output 1016 is a Booleanindicator in which a first value (e.g., true) indicates that the HDRestimation unit 1002 determined the scene to be HDR and in which asecond value (e.g., false) indicates that the HDR estimation unit 1002determined the scene to be non-HDR.

The motion estimation unit 1004 receives one input value 1018 (e.g., thesecond input 510 described with respect to FIG. 5). Processing isperformed similar to that described with respect to the motionestimation unit 610 of the pipeline 600 described with respect to FIG.6. In particular, the motion estimation unit 1004 determines whether theinput value 1018 is higher than a first threshold. If the input value1018 is higher than the first threshold, the motion estimation unit 1004determines that camera motion is detected. If the input value 1018 isnot higher than the first threshold, a SAD is determined between thecurrent and the previous thumbnails. If the SAD is higher than a secondthreshold, the motion estimation unit 1004 determines that scene motionis detected. If the SAD is not higher than the second threshold, themotion estimation unit 1004 determines that no motion is detected. Themotion estimation unit 1004 outputs an output 1020, a Boolean indicator,as a result of the one or two determinations.

The light intensity estimation unit 1006 receives one input value 1022(e.g., the third input 512 described with respect to FIG. 5). Processingis performed similar to that described with respect to the lightintensity estimation unit 616 of the pipeline 600 described with respectto FIG. 6. In particular, the input value 1022 represents a sceneluminance estimated within exposure correction operations. However, theinput value 1022 may in some cases be an ISO value instead of anexposure value. The input value 1022 is compared against a threshold forlight intensity. If the input value 1022 is greater than the threshold,the light intensity estimation unit 1006 determines that the lightintensity is high. If the input value 1022 is less than the threshold,the light intensity estimation unit 1006 determines that the lightintensity is low. If the input value 1022 is neither greater nor lessthan the threshold, the light intensity estimation unit 1006 determinesthat the light intensity is medium.

The output 1016, the output 1020, and the output 1024 may each berepresented as a set of three values, each between 0.0 and 1.0. Thosevalues correspond to a location within a three-dimensional region of thethree-dimensional decision cube. The parameter tuning unit 1010 receivesthe output 1016, the output 1020, and the output 1024 and determines,based on the values included in each of those, the three-dimensionalregion of the three-dimensional decision cube to which the output 1016,the output 1020, and the output 1024 correspond. Data indicative of thatthree-dimensional region is then passed to the mode selection unit 1008.

The mode selection unit 1008 includes a temporal smoothing unit 1028(e.g., the temporal smoothing unit 724 described with respect to FIG.7). The temporal smoothing unit 1028 processes the values correspondingto the three-dimensional region using temporal smoothing filtering, suchas to avoid instabilities. In some implementations, the temporalsmoothing unit 1028 may be external to the mode selection unit 1008. Theselected camera mode 1026 may then be used, such as by an image captureunit (e.g., the image capture unit 408 described with respect to FIG.4), to capture an image.

However, in addition to the temporal smoothing filtering performedagainst the input to the mode selection unit 708, the temporal smoothingunit 1028 in the pipeline 1000 also performs temporal smoothing againstthe output. The temporal smoothing filtering on the output is a kind ofmedian filtering on a window containing past values and works on thelast N unsmoothed output, where N is between 1 and 20, to produce asmoothed output. The smoothed output represents the majority mode of theN values of the temporal smoothing filter window.

After a camera mode is selected, and after the search of thethree-dimensional decision cube for the three-dimensional region, theselected camera mode 1026 is added to a buffer of previously unsmoothedcamera modes of length N. A histogram of those N values is computer toselect the camera mode therefrom having the greatest number ofoccurrences within those N values. That camera mode is the majority modeused as the smoothed value. In some cases, where two or more cameramodes share the majority, the smoothed value is the previous smoothedvalue, such as to prevent oscillations.

In some implementations, the parameter tuning unit 1010 may also be usedto update the three-dimensional decision cube used by the mode selectionunit 1008 to select the camera mode. That is, over time, thethree-dimensional regions of the three-dimensional decision cube maychange in size or position, such as based on the inputs received by thepipeline 1000 (e.g., the input 402 described with respect to FIG. 4).The parameter tuning unit 1010 processes these changes to update thethree-dimensional decision cube accordingly.

In some implementations, the parameter tuning unit 1010 may not belocated before the mode selection unit 1008 in the pipeline 1000. Forexample, the mode selection unit 1008 may directly receive the output1016, the output 1020, and the output 1024. The mode selection unit 1008may then use the parameter tuning unit 1010 to identify the selectedcamera mode 1026. For example, the mode selection unit 1008 can send thethree-dimensional index values of the output 1016, the output 1020, andthe output 1024 to the parameter tuning unit 1010. The parameter tuningunit 1010 may then to query the three-dimensional decision cube for theselected camera mode 1026 according to those three-dimensional indexvalues.

Referring next to FIG. 11, a pipeline 1100 used to implement a sixthtechnique for automated camera mode selection is shown. The sixthtechnique implemented using the pipeline 1100 represents an extension ofthe fifth technique implemented using the pipeline 1000 described withrespect to FIG. 10. The sixth technique introduces tuningrecommendations for use in the automated camera mode selection process.In particular, the sixth technique introduces tuning recommendations forface scores of 0.0 to 1.0 for local tone mapping.

The pipeline 1100 includes an HDR estimation unit 1102, a motionestimation unit 1104, a light intensity estimation unit 1106, and a modeselection unit 1108 (which may, for example, be the HDR estimation unit502, the motion estimation unit 504, and the light intensity estimationunit 506 described with respect to FIG. 5, and the mode selection unit412 described with respect to FIG. 4).

According to the sixth technique, the HDR estimation unit 1102 receivestwo input values 1110 and 1112 (e.g., the first input 508 described withrespect to FIG. 5). Processing is performed similar to that describedwith respect to the HDR estimation unit 602 of the pipeline 600described with respect to FIG. 6. In particular, the HDR estimation unit1102 determines whether a scene for the image to be captured is HDRbased on where the two input values 1112 and 1114 are with respect tothe weighting curves. The HDR estimation unit 1102 outputs an output1116 as a result of that determination. The output 1116 is a Booleanindicator in which a first value (e.g., true) indicates that the HDRestimation unit 1102 determined the scene to be HDR and in which asecond value (e.g., false) indicates that the HDR estimation unit 1102determined the scene to be non-HDR.

The motion estimation unit 1104 receives one input value 1118 (e.g., thesecond input 510 described with respect to FIG. 5). Processing isperformed similar to that described with respect to the motionestimation unit 610 of the pipeline 600 described with respect to FIG.6. In particular, the motion estimation unit 1104 determines whether theinput value 1118 is higher than a first threshold. If the input value1118 is higher than the first threshold, the motion estimation unit 1104determines that camera motion is detected. If the input value 1118 isnot higher than the first threshold, a SAD is determined between thecurrent and the previous thumbnails. If the SAD is higher than a secondthreshold, the motion estimation unit 1104 determines that scene motionis detected. If the SAD is not higher than the second threshold, themotion estimation unit 1104 determines that no motion is detected. Themotion estimation unit 1104 outputs an output 1120, a Boolean indicator,as a result of the one or two determinations.

The light intensity estimation unit 1106 receives one input value 1122(e.g., the third input 512 described with respect to FIG. 5). Processingis performed similar to that described with respect to the lightintensity estimation unit 616 of the pipeline 600 described with respectto FIG. 6. In particular, the input value 1122 represents a sceneluminance estimated within exposure correction operations. However, theinput value 1122 may in some cases be an ISO value instead of anexposure value. The input value 1122 is compared against a threshold forlight intensity. If the input value 1122 is greater than the threshold,the light intensity estimation unit 1106 determines that the lightintensity is high. If the input value 1122 is less than the threshold,the light intensity estimation unit 1106 determines that the lightintensity is low. If the input value 1122 is neither greater nor lessthan the threshold, the light intensity estimation unit 1106 determinesthat the light intensity is medium.

The output 1116, the output 1120, and the output 1124 may each berepresented as a set of three values, each between 0.0 and 1.0. Thosevalues correspond to a location within a three-dimensional region of thethree-dimensional decision cube. The parameter tuning unit 1110 receivesthe output 1116, the output 1120, and the output 1124 and determines,based on the values included in each of those, the three-dimensionalregion of the three-dimensional decision cube to which the output 1116,the output 1120, and the output 1124 correspond. Data indicative of thatthree-dimensional region is then passed to the mode selection unit 1108.

The mode selection unit 1108 includes a temporal smoothing unit 1128(e.g., the temporal smoothing unit 724 described with respect to FIG.7). The temporal smoothing unit 1128 processes the values correspondingto the three-dimensional region using temporal smoothing filtering, suchas to avoid instabilities. The temporal smoothing unit 1128 alsoperforms temporal smoothing against the output, such as described abovewith respect to the temporal smoothing unit 1028 of FIG. 10. In someimplementations, the temporal smoothing unit 1128 may be external to themode selection unit 1108. The selected camera mode 1126 may then beused, such as by an image capture unit (e.g., the image capture unit 408described with respect to FIG. 4), to capture an image.

The mode selection unit 1108 also outputs tuning recommendations 1130.The tuning recommendations 1130 are based on face scores for LTM. A facescore is a value from 0.0 to 1.0 and indicates whether there are facesin an image to capture or not. That is, LTM should adapt tuning where aface score is 1.0 indicating that a big face is in the image, but LTMmay not need to adapt tuning where the face score is 0.0 indicating noface in the image or a face too small or too out of focus. Where thetuning recommendations 1130 reflect a higher value, the area of the facecan be normalized by a display window, such as to provide a more robustzoom and field of view (e.g., wide versus linear) function. A smoothingoperation may also be performed on the tuning recommendations 1130 toavoid oscillations due to face detection instabilities. For example, thesmoothing operation may be expressed asfaceScoreSmoothed=alpha*previousFaceScoreSmoothed+(1−alpha)*currentFaceScore.

Further details of implementations and examples of techniques performedusing the systems and pipelines described with respect to FIGS. 1-11 arenow described. FIG. 12 is a flowchart showing an example of a technique1200 for automated camera mode selection. The technique 1200 can beperformed, for example, using hardware and/or software components of animage capture system, such as the image capture device 100 shown inFIGS. 1A-D or the image capture device 200 shown in FIG. 2. For example,the image capture device 100 or the image capture device 200 may includeone or more software components that process an image captured using animage capture device of the image capture device 100 or the imagecapture device 200, for example, to perform automated camera modeselection, such as described in one or more of the pipeline 400, thepipeline 600, the pipeline 700, the pipeline 800, the pipeline 900, thepipeline 1000, or the pipeline 1100.

In another example, the technique 1200 can be performed using anintegrated circuit. The integrated circuit may, for example, be a fieldprogrammable gate array (e.g., FPGA), programmable logic device (PLD),reconfigurable computer fabric (RCF), system on a chip (SoC), ASICs,and/or another type of integrated circuit. An image processor of theintegrated circuit includes a camera mode selection unit and/or an imagecapture unit (e.g., a processor having one or multiple cores) configuredto execute instructions to perform some or all of the technique 1200.

Although the technique 1200 is described with respect to a series ofoperations, the operations comprising the technique 1200 may beperformed in orders other than those described herein. In someimplementations, the technique 1200 may include additional, fewer, ordifferent operations than those described herein.

At 1202, inputs corresponding to dynamic range, motion, and lightintensity are received. At 1204, HDR is estimated based on the dynamicrange inputs. Estimating the HDR based on the dynamic range inputs caninclude using control points of curves for LUTs to compare values ofdark and bright pixels to one or more thresholds. At 1206, motion isestimated based on the motion input. Estimating the motion can includecomparing an angular speed measured with respect to the motion input toone or more thresholds. At 1208, light intensity is estimated based onthe light intensity input. Estimating the light intensity can includecomparing measurements of the scene luminance for the image to captureto one or more thresholds.

At 1210, the HDR, motion, and light intensity outputs are temporallysmoothed. Temporally smoothing the HDR, motion, and light intensityoutputs can include using those outputs to identify a three-dimensionalregion of a three-dimensional decision cube. For example, the HDR,motion, and light intensity outputs may each be expressed as a set ofvalues indicating a three-dimensional coordinate location. That locationcan be identified within a three-dimensional region of thethree-dimensional decision cube.

At 1212, a camera mode is selected. Selecting the camera mode caninclude identifying a camera mode corresponding to the three-dimensionalregion of the three-dimensional decision cube identified by the temporalsmoothing. At 1214, the selected camera mode is used to capture animage. Capturing an image using the selected camera mode can includeadjusting settings of an image capture device according toconfigurations of the selected camera mode.

Some or all of the technique 1200 may repeat continuously until userinput indicating to capture the image is received. For example, aprocessor or image capture device implementing the technique 1200 maycontinuously perform the operations for estimating the HDR, motion,and/or light intensity and/or for performing temporal smoothing untiluser input indicating to capture the image is received. The user inputmay, for example, be represented by a user of a device configured forcapturing the image interacting with an interface element of the device(e.g., a physical button or a portion of a touch screen).

In some implementations, the technique 1200 can include selecting thesame camera mode for capturing a second image. For example, subsequentto capturing the image using the selected camera mode, user inputindicating to capture a second image may be received (e.g., by the userinteracting with an interface element to capture the second image).Based on aspects of the scene of the image or a time at which the imagewas captured, the same camera mode may be selected for capturing thesecond image.

For example, where the scene of the captured image is similar (e.g.,based on a threshold value) to a scene of the second image to capture,the same camera mode can be selected. In another example, where the userinput indicating to capture the second image is received within athreshold amount of time (e.g., 1 second) after the first image iscaptured, the same camera mode can be selected. Selecting the samecamera mode in either of these ways prevents additional resources frombeing spent to determine a camera mode to use when not much has changedsince the most recent camera mode selection.

In some implementations, the technique 1200 can include determining thatthe scene of the image to capture is dark and selecting a dark settingcamera mode in response. For example, one or more of the inputs for thedynamic range, motion, or light intensity may indicate that the image isto be captured during night time or otherwise in a dark area. In such animplementation, an auto-night camera mode may be selected. Selecting theauto-night camera mode may include bypassing or otherwise ignoringaspects of the technique 1200 that would otherwise be used for selectinga camera mode, for example, the temporal smoothing.

Where certain elements of these implementations may be partially orfully implemented using known components, those portions of such knowncomponents that are necessary for an understanding of the presentdisclosure have been described, and detailed descriptions of otherportions of such known components have been omitted so as not to obscurethe disclosure.

In the present specification, an implementation showing a singularcomponent should not be considered limiting; rather, the disclosure isintended to encompass other implementations including a plurality of thesame component, and vice-versa, unless explicitly stated otherwiseherein. Further, the present disclosure encompasses present and futureknown equivalents to the components referred to herein by way ofillustration.

As used herein, the term “bus” is meant generally to denote any type ofinterconnection or communication architecture that may be used tocommunicate data between two or more entities. The “bus” could beoptical, wireless, infrared, or another type of communication medium.The exact topology of the bus could be, for example, standard “bus,”hierarchical bus, network-on-chip, address-event-representation (AER)connection, or other type of communication topology used for accessing,for example, different memories in a system.

As used herein, the terms “computer,” “computing device,” and“computerized device” include, but are not limited to, personalcomputers (PCs) and minicomputers (whether desktop, laptop, orotherwise), mainframe computers, workstations, servers, personal digitalassistants (PDAs), handheld computers, embedded computers, programmablelogic devices, personal communicators, tablet computers, portablenavigation aids, Java 2 Platform, Micro Edition (J2ME) equipped devices,cellular telephones, smartphones, personal integrated communication orentertainment devices, or another device capable of executing a set ofinstructions.

As used herein, the term “computer program” or “software” is meant toinclude any sequence of machine-cognizable steps which perform afunction. Such program may be rendered in any programming language orenvironment including, for example, C/C++, C#, Fortran, COBOL, MATLAB™,PASCAL, Python, assembly language, markup languages (e.g., HTML,Standard Generalized Markup Language (SGML), XML, Voice Markup Language(VoxML)), as well as object-oriented environments such as the CommonObject Request Broker Architecture (CORBA), Java™ (including J2ME, JavaBeans), and/or Binary Runtime Environment (e.g., Binary RuntimeEnvironment for Wireless (BREW)).

As used herein, the terms “connection,” “link,” “transmission channel,”“delay line,” and “wireless” mean a causal link between two or moreentities (whether physical or logical/virtual) which enables informationexchange between the entities.

As used herein, the terms “integrated circuit,” “chip,” and “IC” aremeant to refer to an electronic circuit manufactured by the patterneddiffusion of trace elements into the surface of a thin substrate ofsemiconductor material. By way of non-limiting example, integratedcircuits may include FPGAs, PLDs, RCFs, SoCs, ASICs, and/or other typesof integrated circuits.

As used herein, the term “memory” includes any type of integratedcircuit or other storage device adapted for storing digital data,including, without limitation, read-only memory (ROM), programmable ROM(PROM), electrically erasable PROM (EEPROM), DRAM, Mobile DRAM,synchronous DRAM (SDRAM), Double Data Rate 2 (DDR/2) SDRAM, extendeddata out (EDO)/fast page mode (FPM), reduced latency DRAM (RLDRAM),static RAM (SRAM), “flash” memory (e.g., NAND/NOR), memristor memory,and pseudo SRAM (PSRAM).

As used herein, the terms “microprocessor” and “digital processor” aremeant generally to include digital processing devices. By way ofnon-limiting example, digital processing devices may include one or moreof DSPs, reduced instruction set computers (RISCs), general-purposecomplex instruction set computing (CISC) processors, microprocessors,gate arrays (e.g., field programmable gate arrays (FPGAs)), PLDs, RCFs,array processors, secure microprocessors, ASICs, and/or other digitalprocessing devices. Such digital processors may be contained on a singleunitary IC die, or distributed across multiple components.

As used herein, the term “network interface” refers to any signal, data,and/or software interface with a component, network, and/or process. Byway of non-limiting example, a network interface may include one or moreof FireWire (e.g., FW400, FW110, and/or other variations), USB (e.g.,USB2), Ethernet (e.g., 10/100, 10/100/1000 (Gigabit Ethernet), 10-Gig-E,and/or other Ethernet implementations), MoCA, Coaxsys (e.g., TVnet™),radio frequency tuner (e.g., in-band or out-of-band, cable modem, and/orother radio frequency tuner protocol interfaces), Wi-Fi (802.11), WiMAX(802.16), personal area network (PAN) (e.g., 802.15), cellular (e.g.,3G, LTE/LTE-A/TD-LTE, GSM, and/or other cellular technology), IrDAfamilies, and/or other network interfaces.

As used herein, the term “Wi-Fi” includes one or more of IEEE-Std.802.11, variants of IEEE-Std. 802.11, standards related to IEEE-Std.802.11 (e.g., 802.11 a/b/g/n/s/v), and/or other wireless standards.

As used herein, the term “wireless” means any wireless signal, data,communication, and/or other wireless interface. By way of non-limitingexample, a wireless interface may include one or more of Wi-Fi,Bluetooth, 3G (3GPP/3GPP2), High Speed Downlink Packet Access/High SpeedUplink Packet Access (HSDPA/HSUPA), Time Division Multiple Access(TDMA), Code Division Multiple Access (CDMA) (e.g., IS-95A, WidebandCDMA (WCDMA), and/or other wireless technology), Frequency HoppingSpread Spectrum (FHSS), Direct Sequence Spread Spectrum (DSSS), GlobalSystem for Mobile communications (GSM), PAN/802.15, WiMAX (802.16),802.20, narrowband/Frequency Division Multiple Access (FDMA), OrthogonalFrequency Division Multiplex (OFDM), Personal Communication Service(PCS)/Digital Cellular System (DCS), LTELTE-Advanced (LTE-A)/TimeDivision LTE (TD-LTE), analog cellular, Cellular Digital Packet Data(CDPD), satellite systems, millimeter wave or microwave systems,acoustic, infrared (i.e., IrDA), and/or other wireless interfaces.

As used herein, the term “robot” may be used to describe an autonomousdevice, autonomous vehicle, computer, artificial intelligence (AI)agent, surveillance system or device, control system or device, and/orother computerized device capable of autonomous operation.

As used herein, the terms “camera,” or variations thereof, and “imagecapture device,” or variations thereof, may be used to refer to anyimaging device or sensor configured to capture, record, and/or conveystill and/or video imagery which may be sensitive to visible parts ofthe electromagnetic spectrum, invisible parts of the electromagneticspectrum (e.g., infrared, ultraviolet), and/or other energy (e.g.,pressure waves).

While certain aspects of the technology are described in terms of aspecific sequence of steps of a method, these descriptions areillustrative of the broader methods of the disclosure and may bemodified by the particular application. Certain steps may be renderedunnecessary or optional under certain circumstances. Additionally,certain steps or functionality may be added to the disclosedimplementations, or the order of performance of two or more steps may bepermuted. All such variations are considered to be encompassed withinthe disclosure.

While the above-detailed description has shown, described, and pointedout novel features of the disclosure as applied to variousimplementations, it will be understood that various omissions,substitutions, and changes in the form and details of the devices orprocesses illustrated may be made by those skilled in the art withoutdeparting from the disclosure. The foregoing description is in no waymeant to be limiting, but rather should be taken as illustrative of thegeneral principles of the technology.

What is claimed is:
 1. An image capture device for automated camera modeselection, the image capture device comprising: a high dynamic range(HDR) estimation unit configured to detect whether HDR is present in ascene; a motion estimation unit configured to detect motion within thescene based on one or more motion inputs; a light intensity estimationunit configured to determine whether a scene luminance for the scenemeets a threshold based on one or more light intensity inputs; aparameter tuning unit configured to output a three-dimensional region ina three-dimensional decision cube based on outputs of the HDR estimationunit, the motion estimation unit, and the light intensity estimationunit; a mode selection unit configured to select a camera mode based onan output of the parameter tuning unit; and an image sensor configuredto detect an image of the scene according to the selected camera mode.2. The image capture device of claim 1, wherein the three-dimensionaldecision cube includes non-overlapping three-dimensional regions fordifferent camera modes.
 3. The image capture device of claim 1, the modeselection unit further comprising: a temporal smoothing unit configuredto perform temporal smoothing filtering against the selected camera modeto mitigate camera mode oscillations.
 4. The image capture device ofclaim 3, further comprising: a buffer configured to hold unsmoothedcamera modes; and the temporal smoothing unit configured to determinethe selected camera mode based on contents of the buffer.
 5. The imagecapture device of claim 3, wherein the three-dimensional decision cubeincludes non-overlapping three-dimensional regions for different cameramodes further comprising: the temporal smoothing unit configured toupdate at least one of a size or position of the three-dimensionalregions in the three-dimensional decision cube.
 6. The image capturedevice of claim 3, further comprising: another temporal smoothing unitconfigured to perform temporal smoothing filtering against the outputsfrom the HDR estimation unit, from the motion estimation unit, and fromthe light intensity estimation unit, wherein the parameter tuning unitis configured to determine the three-dimensional region based on anoutput from the another temporal smoothing filtering.
 7. The imagecapture device of claim 1, wherein the light intensity inputs areexposure values.
 8. An imaging system comprising: a processor, theprocessor configured to: determine a high dynamic range (HDR) level in ascene of a to-be-detected image based on an HDR input; determine amotion level within the scene based on a Sum of Absolute Differences(SAD) between two consecutive motion inputs; determine a scene luminancelevel based on light intensity inputs; and automatically select a cameramode based on a combination of the HDR level, the motion level, and thescene luminance level; and an image sensor coupled to the processor, theimage sensor configured to detect the image according to the selectedcamera mode.
 9. The imaging system of claim 8, wherein the HDR level isbased on a sum of a number of dark pixels detected and a number ofbright pixels detected.
 10. The imaging system of claim 8, wherein theprocessor is further configured to perform the HDR level determination,the motion level determination, and the scene luminance leveldetermination continuously until the image is captured.
 11. The imagingsystem of claim 8, the processor further configured to: determine afirst value type for the HDR level, a second value type for the motionlevel, and a third value type for the scene luminance level; determine alocation in a three-dimensional decision cube based on the first valuetype, the second value type, and the third value type; and select thecamera mode based on the location.
 12. The imaging system of claim 11,the processor further configured to: temporally smooth the first valuetype, the second value type, and the third value type prior todetermining the location in the three-dimensional decision cube.
 13. Theimaging system of claim 12, the processor further configured to:temporally smooth the selected camera mode.
 14. The imaging system ofclaim 8, wherein the light intensity inputs are exposure values.
 15. Amethod for automated camera mode selection, the method comprising:determining high dynamic range (HDR) presence in a scene; detectingmotion presence within the scene; determining scene luminance for thescene; determining a three-dimensional region in a three-dimensionaldecision cube based on outputs of the HDR presence determination, themotion presence detection, and the scene luminance determination;automatically selecting a camera mode based on the three-dimensionalregion determination; and detecting an image using the selected cameramode.
 16. The method of claim 15, wherein the three-dimensional decisioncube includes non-overlapping three-dimensional regions for differentcamera modes.
 17. The method of claim 15, further comprising: temporallysmoothing the selected camera mode to mitigate camera mode oscillations.18. The method of claim 15, further comprising: buffering unsmoothedcamera modes; and determining the selected camera mode based on thebuffered unsmoothed camera modes.
 19. The method of claim 15, whereinthe three-dimensional decision cube includes non-overlappingthree-dimensional regions for different camera modes, the method furthercomprising: updating at least one of a size or position of thethree-dimensional regions in the three-dimensional decision cube. 20.The method of claim 15, further comprising: temporally smoothing theoutputs of the HDR presence determination, the motion presencedetection, and the scene luminance determination.